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SECTION I 



Overview of Confidence Testing 
Introducti on 



When a student takes a test, one question usually arising is the 
extent to which that student's score is a true indication of his ability. 
Oftentimes when he is given another test of similar content or the same 
test at a different time, he will obtain a different score due in part to 
irrelevant and, hopefully, minor variations in the measurement instruments 
and processes. Most testing specialists try to minimize these errors of 
measurement, especially those attributable to the occurrance of guessing. 

As an example of how guessing creates error, suppose the question is 
asked, "What is the capital of North Carolina?" The alternatives given 
are : 



(a) Charlotte (b) Winston-Salem (c) Raleigh (d) Greensboro 

Four students, Adams, Johnson, Washington, and Wilson answer the question 
like this: Adams and Washington both say (c) is correct, Johnson says 

(b) is right, while Wilson marks (a) as being the answer. From the answers 
just given to the multiple choice, we could say that Adams and Washington 
both know the same amount and each knows more about the question than either 
Johnson or Wilson. 

Suppose im reality Washington did know the answer to the question and 
knew the other answers were all wrong. Adam 1, on the other hand, did not 
know whlcn a 3wer was right and guessed at (c), being unable to eliminate 
any of the other answers as possibilities. Wilson knew that it was either 
(a) or (c) but could not make up his mind. He knew (b) and (d) were wrong 
and guessed at (a) over (c). Johnson was sure that (d) was the right answer 
and all the others were wrong. This additional information of the four 
students' confidence in their answers gives a better insight into the students 
knowledge of the subject tested. For example, Washington is right and 
confident and should be rewarded more than Adams who only answered correctly 
because he guessed correctly. Johnson also is sure he's right, but is in 
fact wrong, and should be both strongly penalized and given extra training. 

The uncertainty of Adams and Wilson probably indicates a need for some 
remedial work and merits neither strong reward nor penalty. Methods developed 
during the past years for providing more information from each item in a test 
by asking the examinee to indicate his confidence i.n some way are called 
"confidence tests". Confidence testing provides information that helps 
identify who guesses completely, who can eliminate some of the answers, and 
who is sure of his answer. 

This Handbook is intended for test users as an aid in implementing, 
confidence testing procedures in technical training settings. In addition 
to describing such methods, this Handbook discusses the possible areas of 
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application, the advantages and disadvantages of confidence testing, 
time considerations, and problem areas. In addition, vwo methods are 
recommended termed "Pick-One" confidence testing and "Distribute 100 
Points" confidence testing with complete instructions for testing ^ 

supervisors and examinees in cases where scoring is accomplished by hana 
and by computer. 



Pick-One Confidence Testing 

When using the Pick-One confidence testing format the student is 
first asked to pick the answer he thinks is correct and then mark how 
confident he is that the answer he has chosen is, in fact, correct. The 
degree of confidence he expresses will be described in words, and the score 
he will receive if he is right or wrong will be given so that he knows the 
consequence of his responses. These scores are developed so t e 
will score highest if he gives an accurate estimate of his degree of 
certainty. 

A scoring table for a four-alternative item follows along with four 
examples. The score received depends on whether the answer the examinee 
gives is correct and the amount of confidence assigned to his choice* or 
example, if the examinee picks the correct answer and indicates Very bure, 
he receives 10 points. Had he been wrong, he would have been penalized If 
points . 



Scoring Table for Four -Alter native 


Pick-One 


Confidence Testing 














Not 








Very 




Fairly 


Very 


Not 






Sure 


Sure 


Sure 


Sure 


Sure 


Credit if Right 




10 


9 


7 


3 


0 


Penalty if Wrong 




17 


12 


5 


2 


0 












Not 




Example I 


Answer 


Very 

Sure 


Sure 


Fairly 

Sure 


Very 

Sure 


Not 

Sure 


Chicago is a 












(a) state 

(B) city 

(C) country 
(B) village 




X 


□ 


□ 


□ 


□ 



In the above example the subject was very sure, certain in fact, that the 
answer he marked was correct. He received 10 points for this answer since 
it is correct. 
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Example 2 



Answer 

Which town is not in Nebraska? 



Not 

Very Fairly Very Not 

Sure Sure Sure Sure Sure 



(A) Kearney 

(B) Yankton 

( C) Omaha 

(D) North Platte 




□ □ 




□ □ 



The person answering this example quickly eliminated (C) and (D) • He was 
not sure of his choice between the other two possible answers. He knew that 
it was either Kearney or Yankton but did not know which of the two was 
correct. He picked (B) but marked the middle box. He received ? points for 
this answer. 



Example 3 

Who was the commander 
of the Union Forces at 
the battle of Gettysburg? 



(A) Grant 
(3) Hooker 

ta □ 

The person who answered this question knew that Grant was commanding the Union 
forces in the west at this time so he could be eliminated. He had heard the 
other three names but could not associate them with any battles. He guessed 
at (D) and indicated that he was not very sure of his answer- He lost 2 
points on this question since the answer was (0). 



t Cy Meade 
( D) Burnside 
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□ □ □ 



Example ii 

The capital of Illinois 
is 

(A) Chicago 

(B) Springfield 

□ □ □ $ 

The person who answered this question did not know which one was the answer 
Each alternative looked like it was possible. He guessed at ( D) being the 
right answer but indicated that he was not sure. He got no points. 



Hant-oux 
(D) Decatur 






□ 



Numbers of Alternatives Other Than Four 



The examples given up to this point have been for items with four- 
alternative answers. Although other numbers of alternatives might be used 
for some purposes, their use involves different credits and penalties for 
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the right and wrong answers. The table below gives the credits and losses 
Tor true -false, three -alternative, and five-alternative questions. The 
four-alternative credits and losses are included for completeness. 



TABLE 1 



Certainty Levels 





Very 

Sure 


Sure 


Fairly 

Sure 


Not Very 
Sure 


Not 

Sure 


True-False* 


Credit if Eight 


100 


95 


75 


35 


0 


Penalty if Wrong 


300 


225 


135 


20 


0 


Three Alternative 


Credit if Right 


10 


9 


7 


3 


0 


Penalty if Wrong 


20 


ll* 


7 


2 


0 


Four Alternative 


Credit if Right 


10 


9 


7 


3 


0 


Penalty if Wrong 


17 


12 


5 


2 


0 


Five Alternative 


Credit if Right 


10 


9 


7 


3 


0 


Penalty if Wrong 


15 


11 


5 


1 


0 



*Note that the True-False credits and penalties are given to three figures. 
This is so that the examinee can make a choice between ’’Very Sure" and "Sure" 
that would not automatically be in favor of "Sure", as it would be if only 
two rounded figures were used. 



Distribute 100 Poin ts Confidence Testing 

In the Distribute 100 Points confidence items, the student indicates 
his confidence in each of the alternatives ■. He does this by dividing 100 
points over the answers so that the answer he thinks is best receives the 
largest number of points while the other answers receive fewer points. To 
aid him in deciding how to give points, a scoring table is provided for him 
so he can see how much credit he will receive depending on which alternative 
is, in fact, correct. 

The student is allowed to use only multiples of five, 33 1/3 (one-third 
of a hundred), and 66 2/3 (two-thirds of a hundred) points for reasons of 
ease in scoring and responding. The scoring depends only on the points given 
the correct answer with omitted items given a zero score. The scoring table 
follows with four examples using four alternatives. 
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Points assigned to 
correct alternative 

0 

5 

10 

15 

20 

25 

30 

33 1/3 

35 

U0 

U5 

50 



Points assigned to 
Score correct alternative 



-23 


55 


-12 


60 


- 7 


65 


- h 


66 2/3 


- 2 

0 


70 

75 


1 


80 


2 


85 


2 


90 


3 


95 


a 

5 


100 



S core 

6 

6 

7 

7 

7 

8 

8 

9 

9 

9 

10 



Example 1 



Confidence 



Chicago is a 



(A) state 

(B) city 

(C) country 

(D) village 



(A) O 



(B) /£rp 

(C) _g_ 

(D) O 



The person who answered the example was 100# sure that the answer is (B). 
He could lose 23 points if he is wrong but that did not concern him beca 
he knew he was right and would get 10 points credit. 



Example 2 

■Which town is not in Nebraska? 



(A) Kearney 

(B) Yankton 

( C) Omaha 

(D) North Platte 



Confidence 



(A) 

(B) 

(c) znn 

(D) <9___ 



The person answering this question quickly eliminated ( C) and (D). He was 
not sure of his choice between the other two possible answers. He knew 
that it was either Keamsy or Yankton but did not know which of the two _ 
was correct, so he split the 100 points between them. He received 5 points. 
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Example 3 

Who was the commander of the Union forces at the 
battle of Gettysburg? 

(A) Grant 

(B) Hooker 

(C) Meade 

(D) Burnside 



Confidence 



o 



(A) 

(B) 

(C) 

(D) J£0_ 



_2j2_ 

3o 



The nerson who answered this question knew that Grant was _ c ommanding the 
Union forces in the west at this time so he could be eliminated. He had 
heard the other three names but could not associate them with any battles, 
though he had an idea from something he had read, but could not remember, 
that it might be Burnside. He received 1 point (the answer is ^CJJ. 



Example U 

The capital of Illinois is 



Confidence 



(A) Chicago 

(B) Springfield 
(c) Rantoul 

(D) Decatur 



(A) 

(B.) eZSZ 

(C) 

(D) 



The person who answered this question did not know which one was correct . 
Each alternative looked like it was possible. He spread his points evenly 
and neither gained nor lost credit. 



Numbers of Alternatives Other Than Four 

As in the Pick-One confidence testing case, other numbers of alternatives 
might be used for some purposes. Their use involves different scores being 
assigned to the correct answer. The table below gives scores for true-fal.e, 
three -alternative, and five-alternative questions. The four-alternative 
scores are included for completeness. 
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Two Other Methods 



-x-SCoRule Confidence Testing 

The SCoRule ts a mechanical device that helps the examinee allocate 
his confidence in such a way that the points allocated always add to 100. 
tte defice which is given to the examinee exhibits the scoring system being 
used. Since the device is constructed so that the points must add to -.00 
no check of the points assigned to the examinee is needed and he can reco 
the score indicated on the SCoRule. The examiner needs only to circle the 
score given in the space for the correct answer and add these circled 

for a total score. 

The score scale used by the SCoRule uses zero as minimum, where the 
scoring given in the Distribute 100 Points scoring on the previous pages 
sets the omit score equal to zero which is_ equivalent to the average s o e 
attained by completely random guessing. Th= negative scores ave 
reserved in this Handbook for wrong answers confidently expressec or 
expressions of disbelief in correct answers through the assignment of less 
.an one fourth of the 100 points available (or one over the number of _ 
Alternatives as in Table 1). The Distribute 100 Points, as presented in 
this Handbook, uses a scoring rule similar no that used for SCoRule but 
with the provision for equivalent credit (:.ero) for random J* n 

omitted answers. As a result of this requirement, the scoring tables differ 
for different numbers of alternatives. Hence, no single physical device 
such as a SCoRule could be used for different numbers of alternatives. 
Readers interested in obtaining further information concerning SCoRule 
confidence testing should contact the Air Force Human Resources Laborato y. 
Technical Training Division, Lowry AFB, Colorado. 

Five Stars or None Confidence Testing 

Five Stars or None confidence testing is similar to Distribute ^ 100 
Points confidence testing and SCoRule testing in that weights or P°in ■ s 
are assigned to each alternative of an item. The student is given five 
points or stars to distribute over the alternatives in such a way that more 
poSts are given to the alternative he believes. to be correct than any other 
alternative. The process is identical to distribute 100 points with the 
exception that five points are being distributed rather than 100. The 
scoring system is complex, based on how the points or s ars are a o 
to the various alternatives rather than the points assigned to e 
answer. Readers interested in obtaining further information concerning 
Five Stars or None confidence testing should contact the Air Force Human 
Resources Laboratory, Technical Training Division, Lowry AFB, Colorado. 



SCoRule is a product of the Shuf ord-Massengill Corporation. 
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Relative Value of Each Method 

Each of the four methods given above has definite advantages and 
disadvantages. For example, the task required of the student is easier 
for Pick-One confidence testing than the other methods as it is moso 
similar to traditional multiple-choice testing and thus seems more 
natural to the students. Scoring is not too difficult, ana the student 
is most likely to understand the consequences of his confidence marks. 

The major disadvantage is that scoring depends on the number of alternatives 
to the items which would result in scoring difficulties d f items with varying 
numbers of alternatives are used. Also, confidence inf or.. ui is o ameu. 
only for the alternative believed most likely to be correct ra - - ^ than eac 
alternative . 

The Distribute 100 Points method allows one to examine -lie -tuck ... -j s 
degree of belief for each alternative which allows for greater rrecisj :n 
and is beneficial in providing remedial instruction. On the otne-r harm, 
the task is more difficult for the students and the time require- ci for 
scoring is longer. 

SCoRule testing makes the task of allocating 100 points le,s^ comp— icated 
and allows the examinee to easily determine his possible scores unce ae 
records the actual scores. Hand scoring is easier for the see. r as does 
not have to look up any values on a scoring table. On the ccfcsr hand- the 
ScoRules are devices that must be accounted for and do wear fi a get UM* 
klhen an item is omitted, there seems to be no procedure for scoring and e 
time required for the students to take the test is increased since the 
students need to manipulate the device. 

Five Star s_ or None confidence testing requires the least effort on the 
part of the students of the methods where confidence is given for each, 
alternative. The main disadvantage of this method is the complex scoring 
system which is difficult to understand and use. 

It is recommended that the Pick-One or the Distribute 100 Points 
methods of confidence testing be implemented in any program of confidence 
testing. These methods are recommended because they appear to be less 
cornel ex than either SCoRule or Five Star or None. Both Pick-One and 
Distribute 100 Points provide scoring for omitted items which is not, 
provided by the other methods. Both Pick-One and Distribute 100 Points 
are adaptable to machine scorable answer sheets where. SCoRule . is not. Five 
Stars or None is prohibitive where hand scoring is being utilized but offers 
some promise where tests are scored by computer. 



Selection an d the Confidence Test Scale Scores 

Oftentimes it is necessary to have a minimum score for performance. 

If an examinee scores above that minimum, he might be assigned to a 
particular group or advanced in a training course. On the other hand, if 
he falls below the minimum, he might be assigned to another group or 
recycled. The question then arises of what the minimum score should be if 
confidence testing is usd? Usually, scores are considered _n terms of the 
percentage of items correct. The following table gives tae corresponding 




12 



- 10 - 



confidence test scores for various percentage scores. In confidence 
testing, this score will depend on the number of items m the test so 
thateach entry in the table must be multiplied by the number-of items 
S^he particSar test. The formula for the equivalent confidence score 



IS 



confidence score 



= table value x number of items in the test. 



ent score 


Pick-One 


Distribute 100 


95 


9-3 


9-6 


90 


8.7 


9-2 


85 


8.0 


8.8 


80 


7-3 


8.1* 


75 


6.7 


7-9 


70 


6.0 


7.1* 


65 


5-3 


6.9 


60 


U-7 


6.3 


55 


k'O 


5-7 


50 


3-3 


5-0 



For example, suppose that an examinee takes a final examination for a 
block o/ instruction in technical training where a passing score is 
considered to be 70 percent. If this block examonation is given « a 

confidence test, the score equivalent to 70 P erc ® n ^, rt 8tribute ioo 
is 600 if Pick-One confidence testing is used and 71*0 if Distribute iuu 

Points is used- 



Using Confidence Scores 
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As Diagnostic Aids 

Confidence testing may be a useful device for determining student 
weaknesses in areas where some type of remediation may take place if a 
specified level of performance is not obtained. Consider, for example, 
four student s 1 responses to a test item which has been administered as a 
confidence item and the problem of what type of remediation is required 
r. „ d+nripnt The types of student responses are as follows: one 

student answers ihe item correctly and indicates a high degr ee of confident, 
the second student answers incorrectly with a low degree of confidence, 
another ° student also answers incorrectly but with a high degree of confidence, 
and^the fourth student answers correctly with a low degree of confidence. 

The student who answered correctly with a high degree of confidence 
knows ths answer to the question and is confident, thus there is no need to 
instruct him any further about the specific piece of information being 
tested. On the other hand, the other three students seem to “«it =onsid 
ation for varying degrees of remediation since they answered mcorrecuiyj ^ 
or with little certainty. 
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The student who answered incorrectly with high confidence is sure 
his wrong answer is right, may simply have assimilated an incorrect fact, 
and may have misunderstood a technical process. To the extent that his 
error is factual, he can be shown first why the answer he marked was 
incorrect and then, only after he is convinced of its incorrectness, why 
the keyed answer is in fact correct. This process is a two-stage process 
where the requirement is made that the first stage be completed before the 
second stage can begin. 

The case of the student who places little confidence in his wrong 
answer may be different. He is not sure of the answer he marked and can 
probably be easily convinced that it is wrong thus the time required tc 
carry out stage one is very short. Usually, in this case, one can begin 
by just showing him why the correct answer is in fact correct saving time 
in the remediation process. A similar case may be made for the student who 
answered correctly but with little confidence. In this manner the additional 
information provided by the indication of the student's confidence may be 
used as a guide for providing remediation that is not given using traditional 
multiple-choice test items. However, if this student is not sure simply 
because he does not understand at all, his remediation should prove more 
difficult . Clearly, the nature of his problem should be ascertained before 
his remedial training is completed. 

As can be seen from above, when diagnostic tests are administered as 
confidence tests, the instructor may tailor the remediation practice to the 
individual needs of the attending students in an efficient manner. In this 
way, confidence testing provides a method for individualized remediation. 

It should be noted though that confidence testing procedures do not guarantee 
success in remediation but only provide a guide for conducting remediation. 
For example, a student may respond to items with low certainty as a result of 
indifferences to the military, the instructor, or some other entity rather 
than a lack of knowledge. Problems of this type are not in the spirit of 
this Handbook and thus not discussed. 

Other U ses 

Confidence testing may be used in a number of other ways, such as 

1. Confidence testing may be adaptable to a situation where students 
are administered a test of relatively difficult items. It might serve as 
a device for reducing student anxiety in these situations and provides a 
method for rewarding partial knowledge which would not result on a group 
of difficult multiple -choice items. 

2. Confidence testing might be used to evaluate th? quality of 
various items. Items that receive a large number of wror ^ responses that 
are marked with complete confidence indicate some type of vagueness in the 
items or misunderstanding in teaching. 
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3 . Confidence nesting might be use a to reduce the length of tests 
w. ie at the same tii;.>e retaining the same level of reliability. Many 
studies of confidence testing have shown that when a multiple- choice test 
was administered as a confidence test, the reliability of the test increased. 
If an existing mult:' _u-choice test has a sufficient standard 01 reliabili y 
ar.i the content of tne test is such that the test length can be reduced 
without a corresponding loss of validity, the test might be reduced in leng, h 
yet retain the previous reliability standard by introducing a com idence 
format . 



Advantages and Disadvantages of Usin g Confidence 
” Testing in Technical Training 

The advantages of using confidence testing can be summarized as follows: 

1. Confidence testing may provide the test user with more information 
about a student's level of knowledge than does multiple -choice 
testing. 

2 . Confidence testing may provide a more reliable test than multiple- 
choice testing for the same number of items. 

3 . Confidence testing may allow the instructor to tailor remedial 
presentations to individual student weaknesses. 

ll. Confidence testing requires students to think. over their answers 
more carefully before responding thus increasing their sensitivity 
to the content of the questions. 

The disadvantages of using confidence testing procedures as used in this 
project can be summarized as follows: 

1 . More time is required to administer (about two minutes for each 
fifteen minute regular time period) and score (about twice as 
much as regularly required) confidence tests than multiple -choice 
tests . 

2. jf the test is not sufficiently difficult, most students use only 
the extreme confidence responses. 

3 . If students have had no experience with confidence testing, 
personality factors may contaminate the resulting scores. 

I4 . The technique is still in its experimental stage and its imple- 
mentation requires a change in standard procedure. Hence, it may 
meet resistance from instructors. 

5. The students must know and understand the scoring system used. 
This requires special training time as the use of tne technique 
is not widespread. 
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SECTION II 

Instructions for the Use of Confidence Testing 



STmfir'v isors Instructions fo r Pick-One 
Confidence Testing-H and Scoring 

DISTRIBUTE THE ANSWER SHEETS FIRST . CHECK AND SEE IF EVEKI EXAMINEE HAS 
ANSWER SHEETS AND PENCIL. 

distribute the directions to the examinees, check and see if evehi examinee 

HAS DIRECTIONS. 

distribute the test questions. 

mm aloud the following dibections. answer any questions after YOU read 

THE DIRECTIONS. 



Directions for Mult iple- Choice : Pick-One 

You are advised to use your tone effectively and to work as rapidly as you 
can without losing accuracy. Do not waste your time on questions that are 
too difficult for you. Go on to the other questions and come back 
difficult ones later if you can. 

BE SURE YOU UNDERSTAND THE DIRECTIONS BEFORE ATTEMPTING TO ANSWER ANY 
QUESTIONS . 

YOU ARE TO INDICATE ALL YOUR ANSWERS ON THE SEPARATE ANSW^ SHEET .Nocredit 
will be given for anything written on any other paper. Read the question 

carefully <md rea< under 

heading 6 "ANSWER" ^ ^BE SURE THAT YOU MARK THE ANSWER CLEARLY . 

one answer to each question. If you change your answer, be sure that all 

previous marks are erased completely. 

i j j. „ inHinatp h ow sure vou are that the answer you jnst 
to h te indicated in the boxes to the . right of 
If vou are absolute^ sure your answer is correct and all 
Steele are wrong, mark the box labeled very «ure. 

_-i invi nate anv alternatives and all the alternatives seem equally likely to 
y^mrk IZe box 6 labeled not sure. If you can *U^te ulternatiVBS 
as possibilities but cannot eliminate enough to mark the bo * ** 

mark one of the three intermediate boxes you feel appropriate. A 
^rlnTSble thft tans yon how the test will be scored follows with some 

examples . 

rnTT-c' rn-porn jo BEING SCORED SO THAT YOU CAN DO YOUR BEST ONLY IF YOU INDICATE 
YOUR ITeeIIS ScuISS. DO NOT INDICATE YOU ARE SURFU THAN YOU REALLY ARE 
OR LESS SURE THAN YOU REALLY ARE. 




1G 











Not 






Very 

Sure 


Sure 


Fairly 

Sure 


Very 

Sure 


Not 

Sure 


Credit if Right 


10 


9 


7 


3 


0 


Penalty if Wrong 


17 


12 


5 


2 


0 










Not 




Example 1 

Answer 


Very 

Sure 


Sure 


Fairly 

Sure 


Very 

Sure 


Not 

Sure 


Chicago is a 












(A) state 

(B) city 

( C) country 

( D) village ./5 . 


zf 


• ? 


— 


i 


— 



In the above example the subject was very sure, certain in fact, that the 
answer he marked was correct. He was rignt and so received 10 points. 



Example 2 



Which town is not in Nebraska? 



(A) Kearney 

(B) Yankton 

( C) Omaha 

(D) North Platte 





The person answering this example quickly eliminated ( C) and ( b) • He was 
not sure of his choice between the other two possible answers. He knew 
that it was either Kearney or Yankton but did not know which of the two was 
correct. He picked (B) but marked the middle box. Being correct, he was 
awarded 7 points. 



Example 3 

Who was the commander of the 
Union Forces at the battle 
of Gettysburg? 



(A) Grant 

(B) Hooker 

( C) Meade 

( D) Burnside 



- 7 ) 

ii ii 




The person who answered this question knew that Grant was commanding the 
Union Forces in the west at this time so he could be eliminated. He had 
heard the other three names but couldn't associate them with any battles. 
He guessed at (b) and indicated that he was not very sure of his answer. 
Unfortunately he was wrong, but he only lost 2 points. 
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Not 





Very 

Sure 


Sure 


Fairly 

Sure 


Very 

Sure 


Not 

Sure 


Credit if Right 


10 


9 


7 


3 


0 


Penalty if Wrong 


17 


12 


5 


c 


0 










Not 




Example li 

Answer 


Very 

Sure 


Sure 


Fairly 

Sure 


Very 

Sure 


Not 

Sure 


The capital of Illinois is 












(A) Chicago 

(B) Springfield 

( C) Rantoul 

Id) Decatur J) . 


□ 


□ 


□ 


□ 


3 



The person who answered this question did not know which answer was correct. 
Each alternative looked as though it was possible. He guessed (D) as being 
the right answer but indicated that he was not sure . He received neither 
credit nor penalty . 

If these four examples were taken as a test, the person taking them who marked 
as indicated above would get 15 points, 10 for the first example, 7 for the 
second, would lose two for the third, and would get zero for the fourth. 
Answering helps even when you are not quite sure, but it does not pay to 
overstate your certainty. On Example 1)., if the person said he was very sure 
he would have lost seventeen points and received a negative score. Only^one 
time out of four would he have been lucky enough to get it right. But 
you are sure, say so because it pays. 



Scoring the Pick-One Confidence Testing 

Pick-One confidence testing scoring is different from multiple-choice 
scoring as can be seen from the examples. Scores are set up so a student 
can indicate uncertainty, but risks more as he expresses more certainty. 
Credits and penalties are set up so that he can expect to score best by 
giving his certainty as accurately as possible. 

The score an examinee receives with Pick-One confidence items is equal 
to the sum of the credits for right answers less the sum of the losses for _ 
wrong answers. If a student omits an item, he receives zero credit. Scoring 
is most accurately accomplished if one follows some systematic procedure, 
such as the following: 

1. Circle all the incorrect answers and mark a zero in the margin by 
the omitted items. 

2. Using the scoring table line indicating the penalty for guessing 
answers, mark in the margin for each circled answer a penalty in 
points depending on the amount of uncertainty indicated. 
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3. 



Using the scoring table line indicauing the credit for correct 
answers mark in the margin for each uncircled answer the credit 
in points depending on the amount of uncertainty indicated. 



U. Add the credits and losses separately. 

5. Subtract the total losses from the total credits. 

An example of scoring an answer sheet illustrating each of the four 
previous examples plus an omitted fifth item follows. 



Example of Pick-One Confidence Test Scoring 



Name 



Date 



Course 



Section 



Total of Credits _Z Z 

Very 



1. 


Answer 

"3 


Sure 

. Ed 


2. 


8 


□ 


3- 


(T) 


□ 


u* 


t > 


□ 


5. 




. □ 



Credit if Right 
Penalty if Wrong 



Less Total of Losses 

Not 



Sure 


Fairly 

Sure 


Very 

Sure 


□ 


□ 


□ 


□ 


tlf 


□ 


□ 


□ 


& 


□ 


□ 


□ 


□ 


□ 


□ 






Very 

Sure 

10 

17 



^ Yields a Score of 



Not 

Sure Credits Penalty 

□ .Z2- 

□ __2_ 

□ — s£- 

Ef —Cl. 

□ 

Not 

Fairly Very Not 
Sure Sure Sure Sure 

9730 

12 5 2 0 
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Directions for Students Using Pick-One 
Confidence Testing-Hand Scoring 

pi sections for Multiple- Choice: Pick-One 

You are advised to use your time effectively and to work as rapidly as you 
can without losing accuracy. Do not waste your time on questions that are 
too difficult for you. Go on to the other questions and come back to the 
difficult ones later if you can. 

BE SURE YOU UNDERSTAND THE DIRECTIONS BEFORE ATTEMPTING TO ANSWER ANY QUESTIONS. 

YOU ARE T~ INDICATE ALL YOUR ANSWERS ON THE SEPARATE ANSWER SHEET. No credit 
will be g^ven for anything written on any other paper. Read the question 
caref ully and read each alternative. After you have decided which of the 
suggested answers is most nearly correct , mark that answer on the line under 
heading "ANSWER." BE SURE THAT YOU MARK THE ANSWER CLEARLY. Give only the 
one answer to each question. If you change your answer, be sure that all 
previous marks are erased completely. 

Now, you are asked to Indicate how sure you are that the answer you just 
marked was correct. This is to be indicated in the boxes to the right of 
your first mark. If you are absolutely sure your answer is correct and all 
the other alternatives are wrong, mark the box labeled very sure. If you 
cannot eliminate any alternatives and all the alternatives seem equally 
likely to you, mark the box labeled not sure. If you can eliminate any 
alternatives as possibilities but cannot eliminate enough to mark the box 
labeled very sure, mark one of the three intermediate boxes you feel appro- 
priate. A scoring table that tells you how the test will be scored follows 
with some example s . 

THE TEST IS BEING SCORED SO THAT YOU CAN DO YOUR BEST ONLY IF YOU INDICATE 
YOUR SURENESS ACCURATELY. DO NOT INDICATE YOU ARE SURER THAN YOU REALLY ARE 
OR LESS SURE THAN YOU REALLY ARE. 
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Credit if Right 
Penalty if Wrong 



Not 

Very Fairly Very Not 

Sure Sure Sure Sure Sure 

10 9 7 3 0 

17 12 5 2 0 



Example 1 
Chicago is a 



Answer 



Not 

Very Fairly Very 

Sure Sure Sure Sure 



Not 

Sure 



(A) state 

(B) city 

( C) country 

( D ) village 







□ □ 



In the above example the subject was very sure, certain in fact, that the 
answer he marked was correct. He was right and so received 10 points. 



Example 2 

Which town is not in Nebraska? 

(A) Kearney 

(B) Yankton 

The person answering this example quickly eliminated (C) and (D) • He was 
not sure of his choice between the other two possible answers. He knew 
that it was either Kearney or Yankton but did not know which of the two was 
correct. He picked (B) but marked the middle box. Being correct, he was 
awarded 7 points . 



Example 3 

Who was the commander of the 
Union Forces at the battle 
of Gettysburg? 



(A) Grant 

(B) Hooker 

(C) Meade 

(D) Burnside 




□ □ □ 0 



The person who answered this question knew that Grant was commanding the 
Union Forces in the west at this time so he could be eliminated. He had 
heard the other three names but could not associate them with any battles 
He guessed at (D) and indicated that he was not very sure of his answer. 
Unfortunately he was wrong, but he only lost 2 points. 
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Credit if Right 
Penalty if Wrong 



Example ij. 

The capital of Illinois 



Answer 

is 



(A) Chicago 

(B) Springfield 

(C) Rantoul 

(D) Decatur 




Not 



Very 

Sure 


Sure 


Fairly 

Sure 


Very 

Sure 


Not 

Sure 


10 


9 


7 


3 


0 


17 


12 


5 


2 


0 


Very 

Sure 


Sure 


Fairly 

Sure 


Not 

Very 

Sure 


Not 

Sure 


□ 


□ 


□ 


□ 


X 



The person who answered this question did not know which one was the answer 
Each alternative looked like it was possible. He guessedat CD) being the 
right answer but indicated that he was not sure. He received neither any 
credit or penalty. 



If these four examples were taken as a test the person taking them who marked 
as indicated above would get If? points, 10 for the first example, 7 for the 
second, would lose two for the third, and would get zero for the fourth. 
Answering helps even when you are not quite sure, but it does not pay o 
overstate your certainty. On Example U, if the person said he was very sure 
he would have lost seventeen points and received a negative score. Only one 
time out of four would he have been lucky enough to get it right. But if you 
are sure, say so because it pays. 
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Supervisors Instructions for Pick-One 
Confidence Testing-Machine Scoring 

mchine scorable answer sheets can be nsed to respond to Pick-One items 
and might be desirable should a high volume application arise. Thoug 
scoring cannot be accomplished using traditional mark-sensing equipment, the 

scorable answer sheet could be a useful device for ^cording responses 
in digital form in case more sophisticated processing equipme ... 

for this purpose. These instructions are for the IBM 80$ answer sheet with 
four-alternative test items. To use the machine scorable answer sheet the 
directions are as follows: 

DISTRIBUTE THE ANSWER SHEETS FIRST . CHECK AND SEE IF EVERY EXA.MINEE HAS 
ANSWER SHEETS AND PENCIL. 

DISTRIBUTE THE DIRECTIONS TO THE EXAMINEES. CHECK AND SEE IF EVERY EXAMINEE 
KA.S DIRECTIONS. 

DIST RIB UTE THE TEST QUESTIONS. 

READ ALOUD THE FOLLOWING DIRECTIONS. ANSWER ANY QUESTIONS AFTER YOU REA.D 
THE DIRECTIONS. 

Directions for Multiple-Choice: Pick-One 

You are advised to use your time effectively and to work as rapidly as you 
can without losing accuracy. Do not waste your t me on questions that are 
too difficult for you. Go on to the other questions and come back to the 
difficult ones later if you can. 

BE SURE YOU UNDERSTAND THE DIRECTIONS BEFORE ATTEMPTING TO ANSWER ANY QUESTIONS. 

YOU ARE TO INDICATE ALL YOUR ANSWERS ON THE SEPARATE ANSWER SHEET. No ^edit 
will be Eiven for anything written on any other paper. After you have decided 
which of g the suggested answers is most nearly correct, blacken the^o^^ponding 
space on ^he anXer sheet. BE SURE THAT EACH MRK IS BLACK AND OOmML T FILLS 

THE ANSWER SPACE. Give only one answer to each question. * Y Erasures 

answer, be sure all previous marks are erased completely. Incomplete 

may be read as intended answers. 

Now you are asked to indicate how sure you are that the answer you just 
mar ked was correct. In one of the five boxes to the right of the 
space you are to indicate how sure you are that your answer is correct. For 
instance, the five spaces immediately to the right of item 3 are o 
to indicate yo-ir confidence in your response to item 1, use the spaces for 
item 32 to indicate your confidence in your response to item k, and. so iorth. 
Your second set of answers, if any, should start in the spaces for item 61 
and the corresponding set of indications of confidence should be in the column 

starting with number 91. 
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Very 

Sure 


Sure 


Fairly 

Sure 


Not 

Very 

Sure 


Not 

Sure 




A 


B 


C 


D 


E 


Credit if Right 


10 


9 


7 


3 


0 


Penalty if Wrong 


17 


12 


5 


2 


0 



The table above shows how to indicate how certain yon are, the number of 
points yon receive if yon are right, and the number yon lose if yon are wrong 
If yon are very sure about your answer to item 1, mark alternative A for item 
number 31 on the answer sheet. If yon are right yon will get 10 points and 
if yon are wrong yon will lose 17 points. Another example would be that if 
yon are not very sure of your answer to question number 3 you would mark D 
for item number 33 on the answer sheet and if yon are right you will ge-; J 
-oints— if yon are wrong yon will only lose 2 points. The swrer you inoicate 
the more points yon get if yon a.re right and the more yon lo if you ar« 
.wrong . 



THE CREDITS ARE PLANNED 
CERTAINTY ACCURATELY. 
LESS SURE THAN YOU ARE. 



SO THAI YOU WILL DO BEST IF YOU INDICATE YOUR DEGREE 
DO NOT INDICATE THAT YOU ARE SURER ' 'RAN YOU ARE OR 



Example 1 



Chicago is a 

(A) state 

(B) city 

(C) country 

(D) village 



Answer Sheet 



A B C D E A 

!. II » II H li 31. * 



B C D E 

li li li If 



In the above example the subject was very sure, certain in fact, that the 
answer he marked was correct. He was right and so received 10 points. 



Example 2 



Which town is not in Nebraska? 

(A) Kearney 

(B) Yankton 
(c) Omaha 

(D) North Platte 



Answer Sheet 



ABODE 
2. II 8 II II II 32. 



A B C D E 

1 1 II ( II H 



The person answering this example quickly eliminated (c) and (D). He was 
not sure of his choice between Kearney and Yankton but picked Yankton as 
being more likely and marked the middle box. Being correct, he was awarded 

7 points. 
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Credit if Eight 
Penalty if Wrong 



Not 

Very Fairly Very Not 

Sure Sure Sure Sure Sure 

ABODE 

10 9 7 3 0 

17 12 5 2 0 



Example 3 



Who was the commander of the Union Forces at the 
battle of Gettysburg? 



(A) Grant 
(E) Hooker 
( C) Meade 
(D) Burnside 



ABC 

3. il il ii 



Answer Sheet 



D E A 

£ li 33- 'i 



BODE 

j i It f li 



The per -s or who answered this question knew Grant was commanding the Union 
Forces in -he west at this time so he could be eliminated. He had hea ^ 
the other uhree names but could not associate them with any a es. 
guessed at (D) and indicated that he was not very sure of his answer. 
Unfortunately, he was wrong, but he only lost 2 points. 



Example 1; 

The capital of Illinois is 

(A) Chicago 

(B) Springfield 
( C) Rantoul 

(D) Decatur 



Answer Sheet 



A B C D E A 

h. Il li II il ii 3U- II 



B C D E 

K I! II I 



The person who answered this question did not know 
Each alternative looked like it was possible . He 
right answer but indicated that he was not sure, 
credit or penalty. 



which one was the answer . 
guessed at (D) being the 
He received neither any 



TTTE SPfTRF IS PLANNED SO THAT YOU WILL DO BEST IF YOU INDICATE YOUR BELIEFS 
ACCURATELY. DO NOT GIVE AN ALTERNATIVE MORE POINTS THAN YOU THINK YOU SHOULD 
OR LESS THAN YOU THINK YOU SHOULD. 



This completes the directions as they deal with scoring Pick-One items. 
The user might wish to design another format for the answer sheet which 
includes, for example, a better coordination of the item numbers where the 
confidence responses are made with the item numper where the answer is 
indicated. Such might be desirable for very high volume application. The 
directions here are for use with the answer sheet traditionally used with the 
IBM 805 scoring machine though as indicated above that equipment cannot 
accomplish the scoring. 
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Directions for Students Using Pick-Jne 
Confidence Testing- Machine Scoring 

Ton are advised to use your time effectively and to work as rapidly as you 
can without losing accuracy. Do not waste your time on questions that are 
to" difficult for you. Go on to the other questions and ccme back to the 
difficult ones later if you can. 

3E SURE YOU UNDERSTAND THE DIRECTIONS BEFORE ATTEMPTING TO ANSWER ANY 
QUESTIONS . 

corre sp onding SSl»S«W™*« * BLACK AND 

Sc rr^sMew: 

Incomplete erasures may be read as _atended answers. 

Now you are asked to indicate how sure you are that the t^iswer ywjus^ 
marked was correct. In one of the uve boxes to tne xga _ 1 

soace you are to indicate how sure you are tnat your answer is correc . 

For instance, the five spaces immediately to the right o i em 3 spaces 

used to indicate your confidence in your response to item 1, use the P 
for item 32 to indicate your confidence in your response to jtem 2, and^so 

forth. Your second set of answers, if any, should a should be in 

item 61 and the corresponding set of indications of confidence should be m 

the column starting with number 91* 



Very 

Sure 


Sure 


Fairly 

Sure 


Not 

Vexy 

Sure 


Not 

Sure 


A 


B 


C 


D 


E 


10 


9 


7 


3 


0 


17 


12 


5 


2 


0 



Credit if Right 
Penalty if Wrong 

The table above shows how to indicate how certain you are, the nmb e r ° f 

ioints vou receive if you are right, and. the number you lose if you are wrong 
points you receive ix you ° * to item 1, mark alternative A for item 

If you are very sure about your answer toxtj m 1, m points and 

number 31 on the answer sheet. It you are ngnu you £ 

?f vou are wrong you will lose 17 points. Another example would be ohat if 
jLTreTot very Lre of your answer to question you wouid mark D 

for item number 33 on the answer sheet and if you are right you will ge b 
nointo-inou are Song you will only lose 2 points. The surer you indicate 
the more points you get if you are right and the more you lose if you are 

wrong . 

THE CREDITS ARE PLANNED SO THAT YOU WILL DO BEST IF YOU INDICATE YOUR DEGREE 
OF^ CERTAINTY ACCURATELY. DO NOT INDICATE THAT YOU ARE SURER THAN YOU A 
LEGS SURE THAN YOU ARE. 
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Credit if Right 
Penalty if Wrong 



Sure 



Very 

Sure 

A 

10 

17 



B 



hot 

Fairly ~ery 
Sure lure 

C D 



Not 

Sure 

E 



9 7 3 0 

12 5 2 0 



Example 1 



Chicago is a 

(A) state 

(B) city 

: C) country 
(D) village 



Answer Sheet 



ABODE A 

1. U II \\ ii It 31. I 



B C D E 

li II li a 



In the above example the subject was very sure* certain in facu 5 that the 
answer he marked was correct. He was right and so received 10 points. 



Example 2 



Which town is not in Nebraska? 

(A) Kearney 

(B) Yankton 

(C) Omaha 

(D) North Platte 



Answer Sheet 

ABODE ABODE 

2. 11 | ii 11 11 32. 11 n « 11 ii 



The person answering this example quickly eliminated ( C) and (d)* He was 
not sure of his choice between Kearney and Yankton but picked Yankton as 
being more likely and marked the middle box. Being correct, he was awarded 
7 points. 



Example 3 



Who was the commander of the Union Forces at the 
battle of Gettysburg? 



(A) Grant 

(B) Hooker 

(C) Meade 

(D) Burnside 



ABC 

3. ii a ii 



Answer Sheet 



D E ABODE 

f II 33. 11 11 11 * 11 



The person who answered this question knew Grant was commanding the Union 
Forces in the west at this time so he could be eliminated. He had heard 
the other three names but could not associate them with any battles. He 
guessed at (D) and indicated that he was not very sure of his answer. 
Unfort una tely, he was wrong, but he only lost 2 points. 



Credit if Right 
Penal ~y if Wrong 
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Very 

Sure 

A 



Not 

Fairly Very 
Sure Sure Sure 

BCD 



Not 

Sure 

E 



10 9 7 3 

17 12 5 2 



Example h 

The capital of Illinois is 

(A) Chicago 

(B) Springfield 

(C) Ifa.nmc.ul 

(D) Decatur 



Answer Sheet 



ABODE A 

U. |i ii II II M 3U. il 



B C D E 

n ii ii e 



The person who answered this question did not know whxch one was answer. 
Sch alternative looked like it was possible. He gu< 3sse d at bexng 

Sght answer but indicated that he was not sure. He receivea nexther any 

credit or penalty. 
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Ins true t ions for Distribute 100 Points 
Jidence Testing-Hand Scoring 



DISTRIBUTE THE AITS' 
ANSWER SHEETS AHE : 

DISTRIBUTE THE EIE 
HAS DIRECTIONS. 



SHEETS FIRST. CHECK AND SEE IF EVERY EXAMINEE HAS 
IL. 

DNS TO THE EXAMINEES. CHECK AND SEE IF EVER! EXAMINEE 



DISTRIBUTE THE TUT: QUESTIONS. 

READ ALOUD THE FOlIOFIMG DIRECTIONS. 
DIRECTIONS. 



ANSWER ANY QUESTIONS AFTER YOU READ THE 



Directions for Pis- mte 100 Points 

You are advised t ■ _ae your tine effectively and to work as rapidly as you 
can curacy. Do not waste your tine on questions that are 

too difficult for -cc. Go on to the other questions and come oack to the 
difficult ones later if you can. 

BE SURE YOU UNDERSTAND THE DIRECTIONS BEFORE ATTEMPTING TO ANSWER ANY QUESTIONS. 

YOU ARE TO INDICATE aLL YOUR ANSWERS ON THE SEPARATE ANSWER SHEET. No credit 
22^2 given for anything written on any other paper Read each questx^ 
a earefullv. You should form some belief as to the truun 

of each alternative listed. You are required to ilfsuch 1 

alternative bv distributing 100 points over the possible alternatives in suen 
fw M thartS alta^tive g you believe most likely correct -ceives^most 
number of uoints. *::or example, if you are completely sure ° Y _ n * 

place all 100 point” on the line corresponding to that alternative - 
Se : reminder. P If you cannot eliminate any of the alternatives 
incorrect, place 2$ points on each alternative. You can simplify your task 
by deciding P whicb alternatives you know to be wrong and give themz^ro 
Wor^nu distribute points for alternatives you believe might be correct. 
When you finish, tae largest number of points should be on the alterna ive 
you believe most Xikely to be correct « 

Your scores will be based on the number of points you assign to the correct 
answer. A scoring table is given below to help you understand the scoring. 

TOT P15ir p T .Ps AFF PV.ANNED SO THAT YOU WILL DO BEST IF YOU INDICATE YOUR DEGREE 
“cSi™. DO NOT INDICATE THAT TOO ARE SORER THAN TOO ARE OR 

LESS SURE THAN YOU ARE. 
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Points assigned 



to correct 

alternative Score 



0 -23 

5 -12 

10 - 7 

13 - h 

20 - 2 

23 0 

30 1 

33 1/3 2 



Points assigned 
to correct 
alternative 

33 

U0 

a3 

30 

33 

60 

63 

66 2/3 



Score 

2 

3 

a 

3 

6 

6 

7 

7 



Points assigned 
to correct 



alternative Score 

70 7 

73 8 

8o 8 

83 9 

90 9 

93 9 

100 1 ° 



3 s; aSrKS 

for marking. Read the following examples carefully before beginning. 



Example 1 
Chicago is a 



Confidence 

(A) -4iU- 

(B) 

(c) .. J2— 
(D) . 



(A) state 

(B) city 

(C) country 

(D) village 

ha toS ha Ss rigt and W ould gat 10 points credit. Ha is right and gets 
10 points. 



Example 2 



Which town is not in Nebraska? 



(A) Kearney 

(B) Yankton 

(C) Omaha 

(D) North Platte 



Confidence 

(A) -J~g> 

(B) 

(C) n _ 

(D) O. - 



£ iSjerl^k £ /that 

“ -awss s ’ 

is 3 • 
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Points assigned 
to correct 



alternative Score 



0 -23 

5 "12 

10 - 7 

1 $ ~ U 

20 - 2 

25 0 

30 1 

33 1/3 2 



Points assigned 



to c orre ct 

alternative Score 



35 2 

1*0 3 

1*5 h 

5o 5 

55 6 

60 6 

65 7 

66 2/3 7 



Points assigned 



to correct 

alternative Score 



70 7 

75 8 

60 8 

65 9 

90 9 

95 9 

100 10 



Be sure your points add to 100 . For ease in marking you should use multiples 
of five or 33 1/3 Cone-third of a hundred) and 62 2/3 (two-thirds of a _ _ 

hundred) for marking. Read the following examples carefully before beginning. 



Example 3 

Mho was the commander of the Union forces at the 
battle of Gettysburg? 



Confidence 



(A) Grant 

(B) Hooker 

( C) Meade 

(D) Burnside 



(A) 

(B) 

(C) 

(C) 






3 a 






The person who answered this question knew that Grant was commanding the 
Union Forces in the west at this time so he could be eliminate . leg 

heard the other three names but could not associate them w y * 

though he had an idea from something he had read, but could not remember, 
that it might be Burnside. Actually, (c) is the right answer and since he 
gave (C) thirty points, his score for the item is 1. 



Example 1* 

The capital of Illinois is 

(A) Chicago 

(B) Springfield 

(C) Rantoul 

(D) Decatur 



Confidence 

(A) 

(B) 

(c) 

(D) &SZ 



The person who answered this question did not know which one i^- the answer. 
Each P alternative looked like it was possible. He spread his points evenly 
Tn d neither gained nor lost credit, since the score for a conxidence of 25 
is aero. He would also have received zero had he omitted the item. 



You have noted that you risk more on an alternative the more confidence you 
place on it— putting a lot of points on an alternative gains you a lot if 
you are right and loses a lot if you are wrong. The scoring system 13 
up so the risk is worth it, though. If you are sure, say so because it pays. 
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Scoring the Distribute 100 Points Confidence Test 

This procedure is different than the one used in Pick-One confidence 
tests. There is no need to indicate which items are right and wrong the 
score is based on the response to the correct answer only. The procedure 
is: 



1. Begin by circling the number of points that the student gave the 
correct answer. 

2. For each item* find the number of points in the column headed 
"number of points on the correct answer. 11 Record the corresponding entry 
in the score column in the margin to the right of the item. 

3. Add these numbers together to obtain the total score. 

As an example consider the four examples previously given. It has been 
pointed out that the number of points are 10, 1* and 0 for example 

items 1, 2, 3, and h respectively. The sum of these numbers is 16 and 
that is the score for a person who answered these four items m the way 
indicated. To show the effect of the penalty scores * suppose the con- 
fidence to example 1 had been given as 100 points for alternative (A) and 
zero for the rest. Since (B) is the answer to the item, the score for the 
zero points given to (B) would be -2 3. This would make the item scores 
- 23 y 1* and zero for a total of -17 points. 
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Directions for Students Using Distribute 100 Points 
Confidence Testing-Hand Scoring 



You are advised "to use your time effectively end to ivork ns rapidly as you. 
can without losing accuracy. Do not waste your time on questions that are 
too difficult for you. Go on to the other questions and come back to the 
difficult ones later if you can. 

BE SURE YOU UNDERSTAND THE DIRECTIONS BEFORE ATTEMPTING TO ANSWER ANY QUESTIONS. 

YOU ARE TO INDICATE ALL YOUR ANSWERS ON THE SEPARATE ANSWER SHEET. No credit 
will be given for anything written on any other paper. Read each question 
and each alternative carefully. You should form some belief as to the truth 
of each alternative listed. You are required to record your belief about 
each alternative by distributing 100 points over the possible alternatives 
in such a way that the alternative you believe most likely correct receives 
the most number of points. For example, if you are completely sure of your 
answer , place all 100 points on the line corresponding to that alternative 
and 0 on the remainder. If you cannot eliminate any of the alternatives as 
being incorrect , place 25 points on each alternative. You can simplify your 
task by deciding which alternatives you know to be wrong and give them zero's 
before you distribute points for alternatives you believe might be correct. 

When you finish, the largest number of points should be on the alternative 
you believe most likely to be correct. 



Your score will be based on the number of points you assign to the correct 
answer. A scoring table is given below to help you understand the scoring. 



Points assigned 



to correct 

alternative Score 



0 -23 

5 -12 

10 -7 

15 - b 

20 - 2 

25 0 

30 1 

33 1/3 2 



Points assigned 



to correct 

alternative Score 



■*< 2 

2*0 3 

1*5 U 

5o 5 

55 6 

60 6 

65 7 

66 2/3 7 



Points assigned 



to correct 

alternative Score 



70 7 

75 8 

80 8 

85 9 

90 9 

95 9 

100 io 



Be sure your points add to 100. For ease in marking you should use multiples 
of five or 33 1/3 (one-third of a hundred) and 66 2/3 (two-thirds of a 
hundred) for marking. Read the following examples carefully before beginning. 



THE CREDITS ARE PLANNED SO THAT YOU WILL DO BEST IF YOU INDICATE YOUR DEGREE 
OF CERTAINTY ACCURATELY. DO NOT INDICATE THAT YOU ARE SURER THAN YOU ARE OR 
LESS SURE THAN YOU ARE. 
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Points assigned 
to correct 
■hp.Tnative 


Score 


Points assigned 
to correct 

alternative Score 


Points assigned 
to correct 
alternative 


0 




-23 


35 


2 


70 


5 




-12 


ho 


3 


75 


10 




- 7 


hS 


ll 


80 


i5 




- h 


5o 


5 


85 


20 




- 2 


55 


6 


90 


25 




0 


6o 


6 


95 


30 




1 


65 


7 


100 


33 


1/3 


2 


66 2/3 


7 




Be sure 


vour points add 


to 100. For ease in marking you should use r 


of five 


or 33 1/3 (one-third of a hundred) and 66 


2/3 (two-thirds ox 


for marking. Read the following examples carefully before beginning 


Example 


1 










Chicago 


is a 








Confidence 



Score 

7 

8 

8 

9 

9 

o 

10 



(A) state 

(B) city 

(C) country 

(D) village 



(A) 

(B) / CTO 

(G) 

(D) C* . 



The person who answered the example was 100$ sure that the answer is (B). 

He could lose 23 points if he is wrong but that did not concern him because 
he knew he was right and would get 10 points credit. He is right and gets 
10 Points. 



Example 2 



Which town is not in Nebraska? 



Confidence 



(A) Kearney 

(B) Yankton 

( C) Omaha 

(D) North Platte 



(A.) ^STO 
(B) 

(G) ^ 

(d) 



The person answering this question quickly eliminated (C) and (D). He was 
not sure of his choice between the other two possible answers. He knew 
that it was either Kearney or Yankton but did not know which of the two is 
correct, so he split the 100 points between them. Since Yankton is correct, 
the score is 3 • 
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Points assigned 
to correct 

alternative Score 

0 -23 

5 -12 

io - 7 

15 -it 

20 " 2 

25 0 

30 1 

33 1/3 2 



Points assigned 
to correct 
alternative 

31? 

ho 

U5 

50 

55 

6o 

65 

66 2/3 



Points assigned 
to correct 

Score alternative 



2 

3 

U 

5 

6 
6 
7 
7 



70 

75 

80 

85 

9-> 

95 

100 



Score 

7 

8 

8 

9 

9 

9 

10 



S are 0^3 3 P J)f (o^ttod°of a F hL^)“nrS 1 V3 y ^o.p^nf m a 1 hSdred) 

for marking^ Read the following examples carefully before beginning. 



Example 3 

Who was the commander of the Union forces ax the 
battle of Gettysburg? 

(A) Grant 

(B) Hooker 

(C) Meade 

(D) Burnside 

The person who answered this question He had 

Union forces un the west at h x not associa te them with any battles, 

heard the other three names u ^ could not remember., 

though he had an idea from sometn..ng the’ right answer and since he 

that it might be Burnside . Actually, CC) is the rig 
gave (C) thirty points, his score for the item 



Confidence 

(A) <Q_. 

(B) ^O-. 

(C) 

(D) i£Q_ 



Confidence 






(A) 

(B) 

(c) 






(d) 



Example h 

The capital of Illinois is 

(A) Chicago 

(B) Springfield 

(C) Rantoul 

(D) Decatur 

The person who angered this question did not know which one is^he answer. 
Each alternative looked like it was possibly He =P 5 e “ a h “ n ?°“ nce of 2 | 

ifser“ he Se E ;^d the its,. 

T *“™ "r ^n g Tlo/ k o f To e in?s £ SSSSSi™ 

re/thffis/is i£X S you Se sure, say s^ heoause it pays. 



o 

ERIC 



o r : 
o 
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Supervisors I ns t ruc tions for Di str _bi< te 103 Points 
Confidence Testing-Ma'chi: .e Scaring 

When high volume applications of Distribute 100 Points confidence testing 
is desired, the process can be modified for machine scoring. The following 
instructions utilize the IBM 805 answer sheet. 

DISTRIBUTE THE ANSWER SHEETS FIRST . CHECK AND SEE IF 'WERT EXAMINEE HAS 
ANSWER SHEETS AND PENCIL-. 

DISTRIBUTE THE DIRECTIONS TO THE EXAMINEES. CHECK AND SEE IF EVERY EXAMINEE 
HAS DIRECTIONS. 

DISTRIBUTE THE TEST QUESTIONS. 

READ ALOUD THE FO LL OWING DIRECTIONS. ANSWER ANY QUESTIONS AFTER YOU READ 
THE DIRECTIONS. 

Directions for Distribute 100 Points 

You are advised to use your time effectively and to work as rapidly as you 
can without losing accuracy. Do not waste your time on questions that are 
too difficult for you. Go on to the other questions and come back to the 
difficult ones later if you can. 

BE SURE YOU UNDERSTAND THE DIRECTIONS BEFORE ATTEMPTING TO ANSWER ANY QUESTIONS. 

YOU ARE TO INDICATE ALL YOUR ANSWERS ON THE SEPARATE ANSWER SHEET. No credit 
will be given for anything written on any other paper. Read each question 
and each alternative carefully. You should form some belief as to + he truth 
of each alternative listed. You are required to record your belief about each 
alternative by distributing 5 points over the possible alternatives in such 
a way that the alternative you believe most likely correct receives the most 
number of points. For example, if you are completely sure of your answer, 
place all 5 points on that alternative by marking between the lines designated 
E on the answer sheet. When you finish, the largest number of points should 
be on the alternative you believe most likely to be correct. If you cannon 
eliminate any alternativi 

Your score will be based 
answer, A scoring table 

Points assigned to 
correct alternative 

0 
1 
2 
3 

Be sure you blacken each mark completely. If you decide to change a mark, 
erase completely. Poorly erased marks may be read as answers. 



es, omit the item. 



on the number of points you assign to the correct 
is given below to help you understand the scoring. 



Score 

-23 

- 2 

3 

6 



Points assigned to 
c orrect alternative Score 

k 8 

^ 10 

omit 0 
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Points assigned to 
c orr e ct alternative Score 



0 -23 

1 - 2 

2 3 

3 6 



Points assigned to 

correct alternative 

- — 

5 

omit 



Score 

8 

10 

0 



Be sure you blacken each mark completely* 
erase completely. Poorly erased marks may 



If you decide "to change 
be read as answers. 



a markj 



B£ SURE YOU USE ONLY FIVE POINTS . IF YOU USE MORE YOU WILL BE PENALIZED 



READ THE FOLLOWING EXAMPLES CAREFULLY BEFORE BEGINNING . 



Example 1 




Chicago is a 


(A) state 


1. 

o 


(B) city 


c. • 
*0 


( C) country 


J • 

h • 


(D) village 



Answe r Sheet 



ABODE 



|l U H !' P 

ii )i I) ii ii 

li ii ii ii ') 



In the above example the 
answer he marked (Bi was 



subject was very sure, certain in fact, 
correct. He was right and so received 



that the 
10 points. 



Example 2 



Answer Sheet 



Which town is not in Nebraska? 

(A) Kearney 

(B) Yankton 

(C) Omaha 

(D) North Platte 



ABODE 



5. il 

6. il 
7- ii 

8. il 



I M Ii 11 

II 1 il II 

li ii ii li 

ii it ii ii 



The person answering this example quickly eliminated ^.^^kkton IT 

ItTtZTXt 5SMJKTS S2M 

3 points. 



UOTP* The oenalty for using too many or too few marks will be calculated 
— - by the“imputer. ThS scoring formula used in the four-alternative 



case is 



-6.608 + 16.609 log l0 ( r h^ 



, is found by taking the ratio of the number of points given the 
correct answer h and the total number of points allocated °veral 
alternatives for the particular item being scored. If r. is less 
than or equal to .01, the score is -23.217* 



L h 
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Points assigned to 




Points assigned to 




correct alternative 


Score 


correct alternative 


Score 


0 


-23 


k 


8 


1 


- 2 


5 


10 


2 


3 


omit 


0 


3 


6 




Be sure you blacken each mark completely. 


If you decide to change 


a markj 


erase completely. Poorly 


erased marks may 


be read as answers. 





Example 3 

Itfho was the commander of the Union forces at 
the battle of Gettysburg? 

(A) Grant 

(B) Hooker 
( C) Meade 
(D) Burnside 

The person who answered this question knew Grant was commanding the Union 
forces in the west at this time so he could be eliminated. He had heard 
the other three names but could not associate them with Gettysburg. He 
gave Burnside and Meade each two points and Hooker one. He scored 3 having 
placed two points on Meade ^ the correct answer. 



Answer Sheet 



ABC 



9. i! n 



10 . £ 

11. II 

12. ;'j 



a 



li 

ll 

ll 



D E 

ji \l 
II !i 
II II 
ii li 



Example It 



Answer Sheet 



The capital of Illinois is 

(A) Chicago 

(B) Springfield 

(C) Rantoul 

(D) Decatur 



13- 

lii. 

13- 

16. 



ABODE 

n II II s *> 

g li ll if ll 

II (I /i H H 

II !! II U ll 



The person who answered this question thought Chicago was the state capital 
but had some feeling for Springfield. He gave Chicago four points and 
Springfield one. Since Springfield was correct he receives a score of -2 
but that is better than the score of -23 he would have received had he 
placed all his confidence on alternative (A) . 



THE SCORE IS PLANNED SO THA.T YOU WILL DO BEST IF YOU INDICATE YOUR BELIEFS 
ACCURATELY. DO NOT GIVE AN ALTERNATIVE MORE POINTS THAN YOU THINK YOU 
SHOULD OR LESS THAN YOU THINK YOU SHOULD. 
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Directions for Studer tr Using Distribute 100 Points 
Confidence Testing-Machine Scoring 



You are advised to use your time effectively and to work as rapidly as you 
can without losing accuracy. Do not waste your time on questions that are 
too difficult for you. Go on to the other questions and come back tc 
difficult ones later if you can. 



BE SURE YOU UNDERSTAND THE DIRECTIONS BEFORE ATTEMPTING TO ANSWER ANY QUESTIONS. 



YOU ARE TO INDICATE ALL YOUR ANSWERS ON THE SEPARATE ANSWER SHEET. No c. ^dit 
will be given for anything written on any other paper. Read each question 
and each alternative carefully. You should form some belie, astothetrut 
of each alternative listed. You are required to record your’ belief about each 
alternative by distributing 5 points over the possible alternatives m such 
way that the alternative you believe most likely correct receives the mos 
number of points. For example, if you are completely sure of your a ^wer 
place all 5 points on that alternative by marking between the lines designated 
E on the answer sheet. When you finish, the largest number of points should, 
be on the alternative you believe most likely to be correct. If you cannot 
eliminate any alternatives, omit the item. 



Your score will be based on the number of points you assign to the 
answer. A scoring table is given below to help you understand the 



correct 
scoring . 



Points assigned to 
c orrect alternative 

0 
1 
2 
3 

Be sure you blacken each mark completely. If you decide to change a mark 
erase completely. Poorly erased marks may be read as answers. 

BE SURE YOU USE ONLY FIVE POINTS. IF YOU USE MORE YOU WILL BE PENALIZED. 



Score 

-23 

- 2 

3 

6 



Points assigned to 
correct alternative 

a 

3 

omit 



Score 

8 

10 

0 



READ THE FOLLOWING EXAMPLES CAREFULLY BEFORE BEGINNING. 



THE CREDITS ARE PLANNED 
OF CERTAINTY ACCURATELY. 
LESS SURE THAN YOU ARE. 



SO THAT YOU WILL DO BEST IF YOU INDICATE YOUR DEGREE 
DO NOT INDICATE THAT YOU ARE SURER THAN YOU ARE OR 



Example 1 



Answer Sheet 



Chicago is a 



(A) 


state 


1. 

r \ 


(B) 


city 


d. . 


(c) 


c ountry 


3. 


(D) 


village 


4- 



ABODE 



II 



II 

II 



I I 



li 

I 

li 



ERIC 



O In the above example 
answer he marked was 



the subject was very sure, certain in fact, that the 
correct. He was right and so received 10 points. 
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Points assigned to 

correct alternative Score 



0 -23 

1 - 2 

2 3 

3 6 



Points assigned to 

c orrect alternative Score 

b 8 

5 10 

omit 0 



Be sure you blacken each mark completely. If you decide- to change a mark, 
erase completely. Poorly erased marks may be read as answers. 



Example 2 

■Which town is not in Nebraska? 

(A) Kearney 

(B) Yankton 

( C) Omaha 

(D) North Platte 



Answer Sheet 
ABODE 

Ji i if li li 

6. ii if I i! ^ 

7. ij ll >1 ii 

8. ; i li )i ll ii 



The person answering this example quickly eliminated ( C) and (D). He was 
not sure of his choice between Kearney and Yankton but picked Yankton as 
being more likely. He was awarded a score of 6 for marking Yankton with 
3 points. 



Example 3 

Who was the commander of the Union forces at 
the battle of Gettysburg? 



(A) Grant 


9. 


(B) Hooker 


10. 


( C) Meade 


11. 


(D) Burnside 


12. 



Answer Sheet 



ABODE 

ij ii ii !'• i', 

| |i ii li M 

il I li >i il 

l« I ii H i\ 



The person who answered this question knew Grant was commanding the Union 
forces in the west at this time so he could be eliminated . He had heard 
the other three names but could not associate them with Gettysburg. He 
gave Burnside and Meade each two points and Hooker one. He scored 3 having 
placed two points on Meade, the correct answer. 



Example it 



Answer Sheet 



The capital of Illinois is 



ABODE 



(A) Chicago 

(B) Springfield 

(C) Rantoul 

(D) Decatur 



13- il ii 11 $ *( 

111. I h II 'I il 

l£. ji J r li I I i * 



The person who answered this question thought Cnicago was the state capital 
but had some feeling for Springfiled. He gave Chicago four points and 
Springfield one. Since Springfield was correct he receives a score of -2 
but that is better than the score of -23 he would have received had he 
placed all his confidence on alternative (A). 
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